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ERIC/CRIER READING REVIEW SERIES 



Volume 1 



March, 1968 



To Mdnltor or Not to Monitor: 
Decisions in the Retrieval of Reading 
Research Journal Literature 



Edward G. Summers 
Indiana University 



The ERIC Clearinghouse on Retrieval of Information and Evaluation on 
Reading is a national clearinghouse which collects., organizes, analyzes, 
and disseminates significant research, information, and material on reading 
to teachers, administrators, researchers, and the public. ERIC/CRIER was 
established as a joint project of the International Reading Association and 
Indiana University in cooperation with the Educational Resources Information 
Center of USOE. The Clearinghouse is part of a comprehensive information 
system being developed for the field of ■ education. 
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The ERIC/CRIER Reading Review Series has been created to disseminate 
the information analysis products of the Clearinghouse, Analysis of 
information can take place on a broad continuum ranging from comprehensive 
re\ 7 iews of the state of the knowledge in a given area to bibliographies 
of citations on various topics. Four genres of documents appear in the 
R eading Review Series . The first type includes bibliographies, with 
descriptive abstracts, developed in areas of general interest. . The sec- 
nna himA rrmn-istfi of biblioeranhles of citations, or citations and 

— - w* 

abstracts, developed on more specific topics in reading. The third type 
jprovides short, interpretive papers which analyze specific topics in 
reading using the existing information collection. Tl|e final genre 
includes comprehensive state-of-the-art monographs which critically 
examine given topics in reading over an extended period of time. 

To Monitor or Not to Monitor ; Decisions in the Retrieval cf Reading 
Research Journal Literature describes the development of an objective 
strategy for selection of journal literature in retrieval of reading 
research. A frequency count was made of the articles appearing in the 
Gray summaries of reading research published annually by the Reading 
Research Center of the University of Chicago. The journals for the 
period 1884 to 1985-66 were ranked in terms of productive importance. 

Lists of journals are suggested that would supply three levels of 
coverage; total coverage, 50% coverage, and 80% coverage. Decision 
factors other than sheer quantity of production that must enter into 
the selection of journals to insure adequate coverage of reading research 
are also discussed. 













Several document corpora make up the Master File of materials which 
is organized for search and retrieval in the ERIC/CRIER Clearinghouse on 
Reading. The Gray Summaries comprise one segment of the file and consist 
of over 4,000 references from the published literature ^ich report read- 
ing research. This material is an important link in the information and 
communication network for reading. A recent study by Summers described 
the procedures followed in assembling the Gray bibliographies and trans- 
f farming them for computer storage and retrieval. The study identified 
journals citing reeding research and the network output in journal and 
article production across the total time span of the summaries - approxi- 
mately 1884 to 1966. The statistics verify the rapidly expanding nature 
of information base in reading research. Journals were also classified 
into core, high frequency and peripheral publications based on the rel- 
ative volume of article output over the time span tabulated. 

One goal of ERIC/CRIER is the establishment of a surveillance system 
which will monitor the journal literature and continue to add important 
articles on reading research to the data base* The question of what 
journals should be monitored, out of the hundreds available to education, 
to insure acquisition of a substantial proportion of the reading research 
articles published becomes an extremely important one. The project using 
the Gray. Gummdries provides basic data which is helpful in establishing 



^^^Fidward G. Summers, Storing and Searching Reading Research by 
Computer . Final Report USOE Project OE-5-10-224. December, 1967. 



the relative productive Importance of journals and In making decisions 
on those which it would be most effective to monitor. 



Pattern of Journal Citation In the Gray Stimmarles. 
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to 1966. The results were arrayed In five periods and are presented In 
Tables 1 through 5. Each table presents, In rank order, the most pro- 
ductive journals for each period and the number of articles reported by 
those journals. For each of the five periods the rank order list was 
first split between the values for the ’hnost-clted" journals carrying 
approximately 50% of the articles and the "leas t«> cited*' journals carry- 
ing the remaining half. The results for the five periods ere presented 
In Table 6. The statistics for each of the five periods follow a 
consistent pattern. The comparisons fall roughly Into a 5% to 10%/90% 
to 95% division. Five to ten percent of the total number of journals 
Identified reported approximately 50% of the articles while the other 
50% was reported by the remaining ninety to ninety- five percent of the 
journals - depending on which period Is being examined. For purposes 
of contrast Table 7 reports the results when the rank order list was 
split for the "most-cited" journals carrying approximately 80% of the 
articles and the "least-cited" journals carrying the remaining 20%. The 
statistics for each of the five periods again follow a consistent pattern. 
The comparisons fall roughly Into a 23% to 29%/72% to 76% division. 
Twenty-three to twenty-nine percent of the total number of journals ' 
Identified reported approximately 80% of the articles while 20% was 
reported by the remaining seventy- two to seventy- six percent of the 
journals - depending on which period Is being examined. 
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Table 



Rank order published research literature in reading - 1884-1924 — 1925-1926, 



Journal 



Total no. 1884- 1924- 1925- 

articles 1924 1925 1926 



Elementary School J, 59 47 

J. of Educational Psychology 45 41 

School & Society 38 32 

„.J, of Educational Research 35 26 

Teachers College Record 23 11 

School Review 14 11 

Pedagogical Seminary 13 12 

Elementary School Teacher 11 11 

Chicago Schools J. 86 

American J. of Psychology 7 7 

Ecl^l, Administration & Supervision 7 6 

_JPsychological Clinic 7 7 

J. of Educational Method 5 2 

American School Board J, 43 

Detroit J. of Education 4 4 

Education 4 4 

Psychological Monographs 4 4 

Educational Review 3 3 

English Journal 3 3 

J, of Experimental Pedagogy 3 3 

J. of Experimental Psychology 2 

Midland Schools 2 2 

Psychological Bulletin 2 2 

School World 2 2 

Teachers J, & Abstract 2 

Univ, of Pittsburgh School of Ed. J, 2 
American Education 1 1 

Atlantic Educational Journal 1 1 

British J. of Psychology 1 1 

College Chronical 1 

Educational Research 1 1 

Forum of Education 1 1 

J, of Applied Psychology 1 

J. of Louisiana Teachers Assn. 1 1 

J. of New York State Teachers Assn. 1 1 

Nature Study Review 1 1 

Peabody J. of Education 1 1 

Popular Science Monthly 1 1 

Public School Messenger 1 

Quarterly J. of Speech Education 1 1 

Training School Bulletin 1 1 

Volta Review 1 1 
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Rank order published research literature in reading - 1936-1937 — 1945-1946, 
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Journal Coverage for the Individual and the Library 

The results of the 50% and the 80% analyses are interesting and 
suggest a number of alternatives for both the individual attempting to 
sharpen his own personal monitoring system for acquisition of reading 
research or the information analysis center attempting to establish a 
viable monitoring and acquisition &ystem« Of course the ideal approach 
for both would be to merely subscribe to all the journals identified in 
Table 5 and monitor them accordingly for relevant articles » With some 
surveillance of the overall literature to continue to add peripheral 
journals, almost total acquisition of the relevant material could be 
realized. However, this idealized approach is seldom possible, partic- 
ularly vdien economics is a major factor. The individual, or the infor- 
mation analysis center operating in isolation, cannot afford the luxury 
of subscribing to the large number of journals which publish one or two 
articles every few years just to maintain complete coverage. The analysis 
of the relative productive importance of various journals does provide 
considerable help to the individual acting as his own small scale moni- 
toring and retrieval service. As indicated in Table 6, subscription to 
just seven "most-cited” journals would enable the researcher to have at 
his fingertips better than 50% of the relevant published research liter- 
ature on reading. By adding 26 journals to total the 33 indicated in 
Table 7 better than 80% of the literature would be available. Although 
the financial outlay to add subscriptions to bring the coverage up to 
80% would be beyond the personal finances of most individuals, it would 
not be out of the ordinary for the researcher to expect his library to 









maintain subscriptions to the additional 26 core publications - or the 
total 33 for that matter. With this combination of library and individual 
resources the researcher would actually have access to the total doc^iimentSi 
on a personal or loan basis* for a considerable amount of the published 
research literature in reading. The results of the analysis provide the 
individual vith objective data on which to base his subscription decisions. 
They also enable him to communicate with the library on an objective basis 
in recommending establishment of subscriptions to maintain a useful basic 
collection on reading. 



Journal Coverage in an Information Analysis Center 

Establishing a monitoring and acquisition system for an Information 
analysis center is another matter. Comprehensive coverage becomes 
extremely important but again is usually attainable only as economics 
permit. In producing bibliographies or state-of-the-art and review 
papers such coverage becomes almost mandatory. A major service would be 
performed by such a center by at least announcing those artic.es that 
would not appear in the 50% coverage maintained by the individual or 
even the 80% coverage provided him in concert with the library collection. 
Another factor also enters here. Even though the combined collection of 
the individual and the library might actually provide access to the total 
documents, the information analysis center helps bridge the gap between 
the documents and the user by providing the indispensable service of 
announcing , either on a current awareness basis or through bibliographic 
and review functions, the relevant articles which may be of interest. 

Most ..ctive professionals do not have the time or the resources to system- 




■S, 



atlcally maintain their own surveillance net, except In a small way, 
even though they might have access to the total document once It Is 
Identified for them. The Information center performs an important ad- 
junct role here In essence enabling the researcher to better use what- 
ever collection he maintains Individually or In cooperation with the 
library. 

In light of the previous discussion It would seem that an Infor- 
mation analysis center would want to maintain a complete monitoring and 
acquisition system if possible. The Individual can monitor a small number 
of highly productive journals but the real service of such a center Is in 
bringing the peripheral journal literature to his attention. The 80% 
level lulght be a compromise with total coverage and the 50% level re- 
presents a bare minimum standard. Even with the resources available to 
the analysis center that would perhaps not be available to the Individual 
or library, few centers could operate economically as Individual units 
on the total subscription basis. However, If centers with missions and 
subject areas falling Into the same domain - such as the broad field of 
education - operate In concert then maximum use can be made of the highly 
expensive peripheral literature because there would be substantial prob- 
ability that articles In the low frequency literature not of use In one 
center would be of use In another. Each of the ERIC units now needs a 
fairly precise definition of its core and peripheral llter^vure. A 
central clearinghouse type arrangement can then engage In the subscrli n 
function and distribute journals and articles In accordance with t^e 
subject definitions of the separate units. A highly effective and valu- 
able service to education could be accomplished within the economic 
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constraints now operating. Current exploratory research In progress within 
the ERIC system will examine the feasibility and operational aspects of 
such a concept. There Is every likelihood that such a system will be lm> 
plemented In the near future for the ERIC network. 

Journal Coverage at the 80% Level 

A number of factors In addition to sheer output have to be considered 
In selecting journals to be monitored for an effective surveillance pro- 
gram to cover the journal literature in any field. For example, the rel- 
ative productive importance of various journals over time as well as at 
the Instant the system Is being established has to be considered. For 
purposes of Illustration the following discussion considers the decisions 
to be made in establishing a monitoring and acquisitions system for read- 
ing research literature which would operate at the 80% i.evel of coverage. 
Table 8 presents a composite of the core literature production, by 
periods, for the total time span covered by the Gray Summaries. The 
vertical column for each period indicates the core journals (those pro- 
ducing 80% of the articles reported) for that period and the number of 
articles published. Many of the journals produced articles in periods 
other than those marked but the volume was not of sufficient magnitude 
to classify them as core publications. The 65 core publications have 
been arrayed in seven categories. Category I includes those journals 

which have been listed as core publications for- four or five of the 

» ^ 

periods. These publications would be essential in any listing of current 
core journals for reading. Journals in Category II . those listed as core 
for three decades, should also be included. The English Journal is in- 
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eluded in this category even though its publication during one of the 
periods was :iot high enough to warrant its being listed as core. Category 
III lists journals which have been core publications for the most recent 
two decades. Category IV lists journals which have been identified as the 
core productions for the most recent decade. Both categories should be 
included as current core publications for reading research. 

Category V is an interesting category and illustrates the shift that 
take?, place among journals in terms of relative productivity oi articles. 
This category includes six journals which were core producers for three 
and four decades but whose volume was not of sufficient magnitude In the 
most recent decade to warrant classification as core. All six journals, 
however, were still producing reading articles in the most recent period 
and may have just had a low publication rate for the current decade or 
may be declining in overall productive importance. In any event, it 
would be wise to include them in a current listing of core journals so 
their production can be monitored to determine if they warrant future 
core status. 

Category VI journals are those which had core production the fourth 
period but fell out of the core status in the most recent decade. Some 
of these journals produced articles the most recent decade and may also 
warrant monitoring to determine if their decline is temporary or will 
continue. Category VII includes those journals which were at one time 
of core status but have not achieved th.^t level of productivity for the 
most recent two, three, and four decades. These publications may still 
produce an occasional article on reading but the volume of publication 
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Is relatively low as compared to their previous output. They can defi- 
nitely be classed as peripheral publications for the current period. 

It would appear that the 47 journals in Categories 1 through IV 
would comprise a highly useful list of current core or most productive 
journals for reading research. Monitoring the publication of these 
journals would assure that 75-80 percent of the published research lit- 
erature related to reading would be identified. Judicious selection of 
additional publications in Category VII . and from other high frequency 
and peripheral journals Identified previously, would increase the prob- 
ability substantially that the majority of relevant sources were being 
Included. This core listing is compiled on the basis of volume of 
production for any one journal. Caution must be exercised in using it 

r 

as the sole means^of selecting important journals for reading research. 
The significance of the rank of the volume of publication for any one 
journal Is relative to the volume of publication for all other journals 
tabulated during that period. Thus, lines of demarcation for the three 
categories - core, high frequency, and peripheral - are seldom completely 
absolute and many journals shifted in and out of these categories, based 
on their re^^tive production, for the time period being tabulated. 

In addition, the relative Importance of some single articles from 
low frequency publications must be considered when Identifying the im- 
portant core journals. For example, the AERA Language Arts and Fine Arts 
Review. Item 47 - Category III , had sufficient relative publication 
volume to classify it as core for one period, but not for the most recent 
decade. The Review is done on a three year cycle and has become one of 
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the most significant and useful reviews for reading research. However, 
if core journals were selected on the basis of publication volume alone 
the review would be omitted. Any listing of core publications should 
consider other criteria in addition to*sheer voltnne of publication, and 
the low producing high frequency and peripheral literature should also 
be combed for the extremely significant occasional articles and reviews 
they can contribute. 

It is interesting to note the nature and distribution of the 33 core 
publicaticns identified for the most recent decade. They Illustrate the 
dlversi^ty of interests involved and the interdisciplinary thrust of recent 
research efforts in reading. Journals from domains of education such as 
child development, administration, curriculum, and educational psychology 
are represented. Reports from areas such as psychology, vision, exception- 
ality, communication, library science, and counseling and guidance are also 
included. This diversity is Illustrated further when looking beyond the 
core publications to high frequency and peripheral journals for the most 
recent decade. Many of these publications are of recent origin and will 
probably evolve as continuous core publications in future years. Another 
interesting phenomenon is the increasing volume of research being identi- 
fied in journals published in other countries. The Alberta Journal of 
Educational Research , the Ontario Journal of Educational Research , and 
the British Journal of Educational Psychology were all classified as 
core. Other journals from other countries also appear as high frequency 
and peripheral publications. 

Table 8 also illustrates one other important development relevant 



34 







to production of reading research. The International Reading Association 
started as a professional society during the middle fifties. The three 
journal publications of the Association, the Reading Teacher , the Journal 
of Reading (fbrmerly the Journal of Developmental Reading) and the Reading 
Research Quarterly , reported 291 articles on reading research for 19$6-57 
to 1965-66. This represents approximately 25 percent of the total produc- 
tion of research articles for the ten year period. Thus, IRA has gradually 
assumed a major role in generating ..asearch literature in the field of 
reading. 

Conclusions 

As was previously pointed out, one important element in the Master 
File of materials making up the ERIC/CRIER data base consists of the 
research reports \dilch have appeared in the published journal literature. 

It is essential that techniques be developed to insure that journal lit- 
erature continues to be added to the data base in a systematic fashion. 

The most Important question to be answered in developing an effective 
monitoring and acquisition program for the journal literature is, "What 
combination of journal titles should be monitored to insure adequate 
surveillance of the literature?" This report describes completed re- 
search on the Gray Summaries which Is particularly helpful In providing 
an answer to this question. It is suggested that the degree of cover- 
age may differ depending on purpose and the economics of the situation, 
Three possible levels of coverage are explored: total acquisition, 
acquisition at the 50% level, and acquisition at the 80% level. It is 
recommended that. Ideally, an information analysis center should strive 
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for total coverage with the 50% and 80% levels being admissible 
alternatives. Decision factors other than pure quantity In selecting 
journals are explored and recommendations made as to what journals 
should be monitored to achieve the 80% level of coverage. Individuals, 




libraries, and information analysis centers could use the results of 
this project in establishing monitoring and acquisition programs on 
an objective basis. 
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